-
Notifications
You must be signed in to change notification settings - Fork 86
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Extending AI Actions #2537
Extending AI Actions #2537
Conversation
code_samples/ai_actions/src/AI/Handler/WhisperAudioToTextActionHandler.php
Outdated
Show resolved
Hide resolved
code_samples/ai_actions/src/AI/Handler/WhisperAudioToTextActionHandler.php
Outdated
Show resolved
Hide resolved
docs/ai_actions/extend_ai_actions.md
Outdated
``` | ||
|
||
``` yaml | ||
[[= include_file('code_samples/ai_actions/config/services.yaml', 33, 37) =]] |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
docs/ai_actions/extend_ai_actions.md
Outdated
``` | ||
|
||
``` yaml | ||
[[= include_file('code_samples/ai_actions/config/services.yaml', 38, 45) =]] |
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
5967dee
to
2982a8e
Compare
code_samples/ai_actions/src/Command/ActionConfigurationCreateCommand.php
Outdated
Show resolved
Hide resolved
code_samples/ai_actions/src/AI/Handler/LLaVaTextToTextActionHandler.php
Outdated
Show resolved
Hide resolved
code_samples/ai_actions/src/AI/REST/Input/Parser/TranscribeAudio.php
Outdated
Show resolved
Hide resolved
code_samples/ai_actions/src/AI/REST/Output/Resolver/AudioTextResolver.php
Outdated
Show resolved
Hide resolved
code_samples/ai_actions/src/AI/REST/Output/ValueObjectVisitor/AudioText.php
Outdated
Show resolved
Hide resolved
code_samples/ai_actions/src/Command/AddMissingAltTextCommand.php
Outdated
Show resolved
Hide resolved
code_samples/ai_actions/src/Form/Type/TranscribeAudioOptionsType.php
Outdated
Show resolved
Hide resolved
code_samples/ai_actions/src/Form/Type/TextToTextOptionsType.php
Outdated
Show resolved
Hide resolved
1dc9abf
to
f14c04e
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've slimmed down both this file and the webpack.config.js as you suggest - I agree, it's better in the long run
|
||
private function convertImageToBase64(?string $uri): string | ||
{ | ||
$file = file_get_contents($this->projectDir . \DIRECTORY_SEPARATOR . 'public' . \DIRECTORY_SEPARATOR . $uri); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Shouldn't this use the binary/IO Handler or similar abstract layer to be compatible with DFS or other cluster storage?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for the suggestion! Ok for you to improve this in a follow-up? I agree it would be best to show a best practise here.
I think https://github.com/ibexa/core/blob/4.6/src/lib/IO/IOBinarydataHandler.php is what I might be looking for, but I will confirm that
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please check my comments.
docs/ai_actions/extend_ai_actions.md
Outdated
## Execute Actions | ||
You can execute AI Actions by using the [ActionServiceInterface](../api/php_api/php_api_reference/classes/Ibexa-Contracts-ConnectorAi-ActionServiceInterface.html) service, as in the following example: |
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
docs/ai_actions/extend_ai_actions.md
Outdated
#### Handle input data | ||
Start by creating an Input Parser able to handle the `application/vnd.ibexa.api.ai.TranscribeAudio` media type. |
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
Co-authored-by: julitafalcondusza <117284672+julitafalcondusza@users.noreply.github.com> Co-authored-by: Adrien Dupuis <61695653+adriendupuis@users.noreply.github.com>
code_samples/ change report
|
@julitafalcondusza @adriendupuis thank you for the suggestions! I've applied them in 8010fce and in 82cc5c4 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Few more remarks but nothing mandatory.
|
||
# Extend AI Actions | ||
|
||
By extending AI Actions, you can make regular content management and editing tasks more appealing and less demanding. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe a link back for readers directly arriving here from a search result.
By extending AI Actions, you can make regular content management and editing tasks more appealing and less demanding. | |
By extending [AI Actions](ai_actions_guide.md), you can make regular content management and editing tasks more appealing and less demanding. |
<<<TEXT | ||
Proteins differ from one another primarily in their sequence of amino acids, which is dictated by the nucleotide sequence of their genes, | ||
and which usually results in protein folding into a specific 3D structure that determines its activity. | ||
TEXT |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would inject less white spaces.
<<<TEXT | |
Proteins differ from one another primarily in their sequence of amino acids, which is dictated by the nucleotide sequence of their genes, | |
and which usually results in protein folding into a specific 3D structure that determines its activity. | |
TEXT | |
<<<TEXT | |
Proteins differ from one another primarily in their sequence of amino acids, which is dictated by the nucleotide sequence of their genes, | |
and which usually results in protein folding into a specific 3D structure that determines its activity. | |
TEXT |
You can pass one directly to the `ActionServiceInterface::execute()` method: | ||
|
||
``` php hl_lines="7-8" | ||
[[= include_file('code_samples/ai_actions/src/Command/ActionConfigurationCreateCommand.php', 77, 85) =]] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would have prefer a rewrite of the previous example.
[[= include_file('code_samples/ai_actions/src/Command/ActionConfigurationCreateCommand.php', 77, 85) =]] | |
$action = new GenerateAltTextAction(new Image([$imageEncodedInBase64])); | |
$actionConfiguration = $this->actionConfigurationService->getActionConfiguration('TODO_alt_text'); | |
$output = $this->actionService->execute($action, $actionConfiguration)->getOutput(); |
But no big deal, reader should connect the dot and see how this could have been used in the previous example.
Use the `Ibexa\Bundle\ConnectorAi\Form\FormMapper\ActionConfiguration\ActionHandlerOptionsFormMapper` class together with the `ibexa.connector_ai.action_configuration.form_mapper.options` service tag to make it part of the Action Handler options form. | ||
Pass the Action Handler identifier (`LLaVATextToText`) as the type when tagging the service. | ||
|
||
The Action Handler and Action Type options are rendered in the back office using the built-in Twig option formatter. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe a screenshot of the default rendering of this example form.
* Generated PHP API Reference * Apply suggestions from code review Co-authored-by: julitafalcondusza <117284672+julitafalcondusza@users.noreply.github.com> Co-authored-by: Adrien Dupuis <61695653+adriendupuis@users.noreply.github.com> * Manual review changes --------- Co-authored-by: julitafalcondusza <117284672+julitafalcondusza@users.noreply.github.com> Co-authored-by: Adrien Dupuis <61695653+adriendupuis@users.noreply.github.com>
* Added extending AI documentation * Apply suggestions from code review Co-authored-by: julitafalcondusza <117284672+julitafalcondusza@users.noreply.github.com> Co-authored-by: Adrien Dupuis <61695653+adriendupuis@users.noreply.github.com> * Manual review changes * Generated PHP reference (#2539) * Generated PHP API Reference * Apply suggestions from code review Co-authored-by: julitafalcondusza <117284672+julitafalcondusza@users.noreply.github.com> Co-authored-by: Adrien Dupuis <61695653+adriendupuis@users.noreply.github.com> * Manual review changes --------- Co-authored-by: julitafalcondusza <117284672+julitafalcondusza@users.noreply.github.com> Co-authored-by: Adrien Dupuis <61695653+adriendupuis@users.noreply.github.com> * Regenerated reference --------- Co-authored-by: julitafalcondusza <117284672+julitafalcondusza@users.noreply.github.com> Co-authored-by: Adrien Dupuis <61695653+adriendupuis@users.noreply.github.com>
This PR builds upon the work done in #2473 - and adds the "how to extend" part of the doc, together with references.
I've included the generated PHP API Reference (this is stolen from #2447) - to avoid polluting this PR with too many files the generated reference is extracted to a separate PR (and all the preview links link there).
General preview: preview
New pages or sections:
Testing the use cases locally:
Setup
Run in a project:
git clone https://github.com/ibexa/documentation-developer -b IBX-8689-extending cp -R documentation-developer/code_samples/ai_actions/ . merge the ibexa_admin_ui.yaml and webpack.config.js changes together with the existing content composer run post-install-cmd
Use case 1
php bin/console app:add-alt-text
The image should have alt text generated automatically
Goal: teach readers how to use the AI PHP API
Use case 2
LLaVATextToText
HandlerGoal: teach readers how to create a custom Handler (including Action Handler Form options)
Use case 3
whisper
eztext
field withtranscript
identifier to the File Content TypeGoal: teach readers how to create a custom Action Type, including:
TODO:
Follow up: fix the PHP reference errors- done in https://github.com/ibexa/connector-ai/pull/104Checklist